There are a number of different enhancements that would be desirable to be performed on video. A number of these enhancements, however, would require manual editing techniques beyond the skill of a user. For example, a user may take a video of his or her spouse on the beach, and may wish to edit out one or more strangers walking into the frame. Currently this requires significant artistic and computing skill, as well as time, on the part of the user to select the portions of each frame of the video to remove, remove those selected sections, and blend the background images to fill in the removed sections to create a seamless look. Since there may be many frames in each second of video, this may result in the need to edit many different frames. It would be beneficial if there were a way to automate this process in an efficient manner.